Bencher

🚨 2 ALERTS: Threshold Boundary Limits exceeded!

Bencher

Bencher

🚨 1 ALERT: Threshold Boundary Limit exceeded!

Bencher

🚨 8 ALERTS: Threshold Boundary Limits exceeded!

How would this work in practice? Lets say I added a new breaking change, will the PR CI keep failing until I change the version accordingly?

The semver check in our CI pipeline ensures that all modifications to the codebase comply with semantic versioning rules. If a breaking change is introduced without the corresponding version increment, the CI will fail.

IIUC we should run this only on versioned branches. I think our release branch is main, so maybe we should run this only when a PR base is main ? Not sure though, Ill let @Fi3 chime in as he opened the issue

here's a summary of our current strategy for semver quality control (which is admitedly suboptimal, so this contribution is very desired)

we have check-versioning-lib-release.sh which is called from .github/workflows/release-libs.yaml every time we are trying to merge from dev to main

if the script returns 1, CI breaks and merging from dev to main is blocked

the script will return 1 if:

• some crate was changed since the latest release
• its version wasn't bumped

basically this is a safety check for releases, but implemented in a very suboptimal way

one of the undesired consequences is that even small cosmetic changes (e.g.: PR #915) impose that crates versioning must be bumped, which is not ideal

in summary, whenever we're merging from dev to main, our goal is:

• breaking changes to any of the crates published to crates.io (list can be found at .github/workflows/release-libs.yaml) must follow SemVer 2.0 (as described in RELEASE.md)
• non-breaking changes (e.g.: README.md, code formatting, docs, etc) are not taken into consideration

I believe we will be able to achieve this vision by replacing check-versioning-lib-release.sh and leveraging cargo semver-checks over this new CI .github/workflows/semver-check.yaml

Thanks @plebhash thats very helpful. @Shourya742 So I think you are half way through, lets just replace the old check with the new one as @plebhash mentioned.

Note here: cargo-semver relies on the result of rustdoc in order to notify about semver breaches, i.e. it reads the docs we write and emit notifications accordingly. As our docs are not great yet, cargo-semver might not produce the results we want. Nonetheless, I think we should add it, use it as an additional check, and once we have robust docs we can rely on it solely. cc @Fi3

Note here: cargo-semver relies on the result of rustdoc in order to notify about semver breaches, i.e. it reads the docs we write and emit notifications accordingly. As our docs are not great yet, cargo-semver might not produce the results we want. Nonetheless, I think we should add it, use it as an additional check, and once we have robust docs we can rely on it solely. cc @Fi3

in other words: this is currently blocked by #845

cargo-semver is designed to ensure semantic versioning (semver) compatibility by analyzing the public API of a Rust crate. . Importantly, cargo-semver does not rely on rustdoc or the quality of the documentation to detect these breaches. In our case, because the jd-server and jd-client crates in roles are not published on crates.io, we utilize the cargo-semver-checks --baseline-root . command. This command allows us to specify the local directory as the baseline for semver checks, enabling us to manage version compatibility internally for our unpublished crates.

Do we want to add this workflow to the .githooks/pre-push script?

Note here: cargo-semver relies on the result of rustdoc in order to notify about semver breaches, i.e. it reads the docs we write and emit notifications accordingly. As our docs are not great yet, cargo-semver might not produce the results we want. Nonetheless, I think we should add it, use it as an additional check, and once we have robust docs we can rely on it solely. cc @Fi3

in other words: this is currently blocked by #845

cargo-semver is designed to ensure semantic versioning (semver) compatibility by analyzing the public API of a Rust crate. . Importantly, cargo-semver does not rely on rustdoc or the quality of the documentation to detect these breaches. In our case, because the jd-server and jd-client crates in roles are not published on crates.io, we utilize the cargo-semver-checks --baseline-root . command. This command allows us to specify the local directory as the baseline for semver checks, enabling us to manage version compatibility internally for our unpublished crates.

https://github.com/obi1kenobi/cargo-semver-checks/blob/main/README.md#how-is-cargo-semver-checks-similar-to-and-different-from-other-tools (third paragraph)

--baseline-root based on the docs this flag is used for getting the older version number

Note here: cargo-semver relies on the result of rustdoc in order to notify about semver breaches, i.e. it reads the docs we write and emit notifications accordingly. As our docs are not great yet, cargo-semver might not produce the results we want. Nonetheless, I think we should add it, use it as an additional check, and once we have robust docs we can rely on it solely. cc @Fi3

in other words: this is currently blocked by #845

cargo-semver is designed to ensure semantic versioning (semver) compatibility by analyzing the public API of a Rust crate. . Importantly, cargo-semver does not rely on rustdoc or the quality of the documentation to detect these breaches. In our case, because the jd-server and jd-client crates in roles are not published on crates.io, we utilize the cargo-semver-checks --baseline-root . command. This command allows us to specify the local directory as the baseline for semver checks, enabling us to manage version compatibility internally for our unpublished crates.

https://github.com/obi1kenobi/cargo-semver-checks/blob/main/README.md#how-is-cargo-semver-checks-similar-to-and-different-from-other-tools (third paragraph)

--baseline-root based on the docs this flag is used for getting the older version number

not sure that I get the issue here. It say that the default option is to use the version of what is published on crates.io

Note here: cargo-semver relies on the result of rustdoc in order to notify about semver breaches, i.e. it reads the docs we write and emit notifications accordingly. As our docs are not great yet, cargo-semver might not produce the results we want. Nonetheless, I think we should add it, use it as an additional check, and once we have robust docs we can rely on it solely. cc @Fi3

in other words: this is currently blocked by #845

cargo-semver is designed to ensure semantic versioning (semver) compatibility by analyzing the public API of a Rust crate. . Importantly, cargo-semver does not rely on rustdoc or the quality of the documentation to detect these breaches. In our case, because the jd-server and jd-client crates in roles are not published on crates.io, we utilize the cargo-semver-checks --baseline-root . command. This command allows us to specify the local directory as the baseline for semver checks, enabling us to manage version compatibility internally for our unpublished crates.

https://github.com/obi1kenobi/cargo-semver-checks/blob/main/README.md#how-is-cargo-semver-checks-similar-to-and-different-from-other-tools (third paragraph)
--baseline-root based on the docs this flag is used for getting the older version number

not sure that I get the issue here. It say that the default option is to use the version of what is published on crates.io

The question is how it knows there is a major breaking change between lets say 1.0 and 1.1 and notify about it. its doing so by inspecting the docs.

@jbesraa Yes, the semver check analyzes the documentation to understand the previous API behavior and detect any breaking changes in the current version. If you introduce a breaking change, the PR CI will fail until you update the version number accordingly, usually by incrementing the major version.

@jbesraa Yes, the semver check analyzes the documentation to understand the previous API behavior and detect any breaking changes in the current version. If you introduce a breaking change, the PR CI will fail until you update the version number accordingly, usually by incrementing the major version.

Yup. which makes this not great for this project(at this point) because we dont have(enough) docs and the API is not stablised yet

@jbesraa Yes, the semver check analyzes the documentation to understand the previous API behavior and detect any breaking changes in the current version. If you introduce a breaking change, the PR CI will fail until you update the version number accordingly, usually by incrementing the major version.

Yup. which makes this not great for this project(at this point) because we dont have docs and the API is not 'stablised' yet

I don't know the internals of it. But rust doc are automatically generated, and you get an entrance for everything that is exported by the library. I'm pretty sure that it do not need us to write actual docs in order to work.

API is stabilized and if we change it we MUST updated the version accordingly.

@jbesraa , that's precisely the point. We've already done a major release (version 1), and the APIs should not change. If there is any breaking change, a version bump is necessary. However, I believe @plebhash might be the right person to ask about this.

@jbesraa , that's precisely the point. We've already done a major release (version 1), and the APIs should not change. If there is any breaking change, a version bump is necessary. However, I believe @plebhash might be the right person to ask about this.

any patch / bug that do not introduce new feature or remove old one MUST be a patch version bump
any new feature MUST be a minor version bump
anything that break compatibility MUST be a majior version bump

This is very important, there are other project out there that use SRI, and since cargo when building a project follow semver, if we do not do it will have build errors!

Sorry maybe I am not explaining myself here

What iam trying to say is that this tool relies on rust-docs(we should write the docs) and while we dont have enough/good docs we might get alot of false alarms because the code is ahead of the code docs..

Sorry maybe I am not explaining myself here

What iam trying to say is that this tool relies on rust-docs(we should write the docs) and while we dont have enough/good docs we might get alot of false alarms because the code is ahead of the code docs..

docs are automatically generated you don't need to write them

I should point out that I'm not 100% confident on what I said here: #985 (comment) so I hid the comment to avoid making things more confusing than necessary

I'd like to ask to everyone (especially @jbesraa ) to take this comment with a grain of salt, it might be the case that we're not necessarily blocked by #845

it seems there's some confusion with regards to how cargo-semver-checks works and how it will fill our needs, and unfortunately the project's FAQ on the README doesn't seem to be fully answering all our questions.

I'm taking some time to study the following resources, which hopefully will allow for more well-informed decisions:

• https://www.youtube.com/watch?v=VArNQtYBC6Y
• https://docs.google.com/presentation/d/1gVYM9-YLrBmRYSuHqW8roE5Rt2FLAS0cijjgFU7ztxY/edit?usp=sharing